As described in Sectioná7.4.4, “RDP authentication”, VirtualBox supports arbitrary external modules to perform authentication with its VRDP servers. When the authentication method is set to "external" for a particular VM, VirtualBox calls the library that was specified with VBoxManage setproperty vrdpauthlibrary. This library will be loaded by the VM process on demand, i.e. when the first RDP connection is made by an external client.

External authentication is the most flexible as the external handler can both choose to grant access to everyone (like the "null" authentication method would) and delegate the request to the guest authentication component. When delegating the request to the guest component, it will still be called afterwards with the option to override the result.

A VRDP authentication library is required to implement exactly one entry point:

#include "VRDPAuth.h"

/**
 * Authentication library entry point. Decides whether to allow
 * a client connection.
 *
 * Parameters:
 *
 *   pUuid            Pointer to the UUID of the virtual machine
 *                    which the client connected to.
 *   guestJudgement   Result of the guest authentication.
 *   szUser           User name passed in by the client (UTF8).
 *   szPassword       Password passed in by the client (UTF8).
 *   szDomain         Domain passed in by the client (UTF8).
 *
 * Return code:
 *
 *   VRDPAuthAccessDenied    Client access has been denied.
 *   VRDPAuthAccessGranted   Client has the right to use the
 *                           virtual machine.
 *   VRDPAuthDelegateToGuest Guest operating system must
 *                           authenticate the client and the
 *                           library must be called again with
 *                           the result of the guest
 *                           authentication.
 */
VRDPAuthResult VRDPAUTHCALL VRDPAuth(
    PVRDPAUTHUUID pUuid,
    VRDPAuthGuestJudgement guestJudgement,
    const char *szUser,
    const char *szPassword
    const char *szDomain)
{
    /* process request against your authentication source of choice */
    return VRDPAuthAccessGranted;
}

A note regarding the UUID implementation of the first argument: VirtualBox uses a consistent binary representation of UUIDs on all platforms. For this reason the integer fields comprising the UUID are stored as little endian values. If you want to pass such UUIDs to code which assumes that the integer fields are big endian (often also called network byte order), you need to adjust the contents of the UUID to e.g. achieve the same string representation. The required changes are:

Using this conversion you will get identical results when converting the binary UUID to the string representation.

The second arguments contains information about the guest authentication status. For the first call, it is always set to VRDPAuthGuestNotAsked. In case the function returns VRDPAuthDelegateToGuest, a guest authentication will be attempted and another call to the method is made with its result. This can be either granted / denied or no judgement (the guest component chose for whatever reason to not make a decision). In case there is a problem with the guest authentication module (e.g. the Additions are not installed or not running or the guest did not respond within a timeout), the "not reacted" status will be returned.